Robot Painter: High-Level Planning Based on Visual Perception

Recently, many areas of research on humanoid robots have been studied, such as motion control, man-machine interfaces, artificial intelligence (AI), and so on. Among them many research projects have tried to create artist robots, with the common objective of exploring new sensing, artificial intelligence, and manipulation techniques. The research described in this thesis explores new vision and manipulation techniques through painting tasks. The ultimate goal is to create a robot painter that has capabilities similar to those of human artists. Regarding vision, the key problems of 2D/3D object segmentation, color perception, orientation mapping, and geometric edge processing are directly addressed by our method. This research focuses, first, on an effective 2D segmentation scheme using local and global classifiers. Our proposed method can effectively deal with a foreground cut, multiple cuts, and cut before matting. Then it is shown how to exploit normal stereo cameras to roughly extract the object automatically, based on 3D background subtraction and other vision techniques, and how to use our 2D segmentation to extract the object area correctly. The robot must analyze color distribution of the object to select the best set of colors to use. Normally, clustering colors face the tendency to produce colors with low contrast. We solve this problem by incorporating two clustering methods: maximum distance clustering and K-means. Then, in order to draw brush strokes meaningfully, the robot senses the orientation of the object. To smooth the orientation of the whole object, we apply global orientation that exploits the radial basis function to generate a style similar to Van Gogh, for instance, for the entire object. Furthermore, some human artists usually use edges to enhance their paintings. Technically, many researchers use gradient information to represent edges of objects. However, this would be affected by the color information on the surface. Hence, we decided to use 3D geometric edges as an input. We then extract 2D edges from the 3D model. Finally, the 2D edges are processed into brush strokes. We show how to apply these methods to high-level manipulation using a robot platform that consists of two arms and multi-fingered hands.